CALYPSO: A Novel Software System for Fault-Tolerant Parallel Processing on Distributed Platforms

The importance of adapting networks of workstations for use as parallel processing platforms is well established. However, current solutions do not always address important issues that exist in real networks. External factors like the sharing of resources, unpredictable behavior of the network, and failures, are present in multiuser networks and must be addressed. Calypso is a prototype software system for writing and executing parallel programs on non-dedicated platforms, based on COTS networked workstations, operating systems, and compilers. Among notable properties of the system are: (1) simple programming paradigm incorporating shared memory constructs and separating the programming and the execution parallelism, (2) transparent utilization of unreliable shared resources by providing dynamic load balancing and fault tolerance, and (3) e ective performance for large classes of coarse-grained computations. We present the system and report our initial experiments and performance results in settings that closely resemble the dynamic behavior of a \real" network. This research was partially supported by the National Science Foundation under grant numbers CCR-91-03953, CCR-9411590, and CCR-95-05519. yFor additional publications and information about the Calypso project, see the URLs: ftp://cs.eas.asu.edu/pub/calypso/ and ftp://cs.nyu.edu/pub/calypso/ z251 Mercer St., New York, NY 10012-1185, (212) 998-3350, [email protected]. ASU, Tempe, AZ 85287-5406, (602) 965-5583, [email protected]. {251 Mercer St., New York, NY 10012-1185, (212) 998-3101, [email protected]. Under varying work-load conditions, resource availability and process failures, the e ciency of the test program we present ranged from 84% to 94% benchmarked against a sequential program. 1 Goals and Properties of Calypso Networks of workstations exist in many organizations, and their number is growing rapidly. These machines are mostly idle. Thus, it is very attractive and cost-e ective to utilize this frequently wasted resource. Given the previous reasons, then why is it that programs that utilize networks of workstations have not proliferated? A major reason is that the cost to harness this power is too high. That is, although networks of workstations are a good value in terms of raw computing power (meaning hardware), the cost to harness this power (meaning software development) still remains high and unattractive. There are systems to utilize this hidden power, but they often do not address some important issues. For example they require extensive changes to the programming model, or they are unable to handle the separation of programming and execution parallelism, or they cannot deal with failures, or they lack adequate load distribution and balancing to account for slow and fast machines. The challenging problems in providing a satisfactory environment for parallel processing on networks of workstations are well known. Such problems include programmability, high-performance, scalability, load balancing, and fault-masking. We have addressed most of these in the design and the implementation of Calypso. In Proc. 4th IEEE Intl. Symp. on High Performance Distributed Computing, August 1995 2 Furthermore, as the commercial and the administrative realities prohibit the vast majority of users from buying special purpose hardware and running \private" operating systems, Calypso utilizes standard hardware and standard software. The current prototype runs under SunOS, and the system has been designed and implemented to be portable. We expect ports to run on most Unix-based operating systems as well as Windows NT. Calypso is a prototype system for writing and executing parallel programs on multiuser networks. It it unique among other systems in that it provides the following features through a uni ed set of solutions. Ease of Programming: The programs are written in Calypso Source Language (CSL). CSL is C++ with added constructs to express parallelism. It is based on the shared memory model and it is very simple to learn and use. The programmer does not have to partition the data, or to specify how to synchronize or move it among the workstations. Separation of Programming Parallelism from Execution Parallelism: The parallelism inherent to a problem is independent of the number machines it will run on. So, why should a parallel program be tied to the number of available workstations|a number that in most cases is unpredictable and transient? This is a common weakness in many systems, and adds complexity to an already di cult program development. The mapping between program parallelism and execution parallelism is transparent in Calypso. Dynamic Load Balancing and Fault Tolerance: Calypso automatically distributes the work-load depending on the dynamics of the participating machines. Faster machines are not blocked waiting for slower machines to nish their \work assignments"|faster machines overtake the slower ones. In addition, Calypso executions are resilient to failures. All processes, other than a speci c designated \manager" process, can fail at any point without a ecting the correctness of the computation. In contrast with other fault-tolerant systems, there is e ectively no additional cost associated with this feature in the absence of failures. This is con rmed by our initial observation that the performance of Calypso is comparable to other non-fault-tolerant systems. High Performance: While providing the features listed above, our initial experiments indicate that the overhead is small, and a large class of coarse-grained computations can bene t from Calypso. The paradigm that Calypso embodies in a working system, was rst described in [26]. That paper details a methodology for instrumenting general parallel programs to automatically obtain their fault-tolerant counterparts. While the solutions in [26] were formulated in the context of synchronous faults, they were later applied in [24] to a certain variant of asynchronous behavior (as does Calypso). A uni ed set of mechanisms, eager scheduling and collating di erential memory, is used to provide the functionality of Calypso. The idempotence property is fundamental in Calypso: a code segment can be executed multiple times (with possibly some partial executions), with exactly-once semantics. The importance of idempotence, and the utilization of the eager scheduling to take advantage of it, was discovered in [26] in an abstract context. (The term \eager scheduling" itself was coined recently.) Eager scheduling is a mechanism for assigning concurrently executable tasks to the available machines. Any machine can execute any \enabled" task independent of whether this task is already under execution by another machine. As a consequence, free machines end up doing more work than loaded ones, leading to a balanced system. Secondly, computations do not stall while dealing with system's asynchrony and faults. Thirdly, newly available machines can transparently be integrated into an executing computation. And nally, any of the machines that are \helping out" the parallel computation can fail or slow down at any time. The mechanism of collating di erential memory provides logical coherence and synchronization while In Proc. 4th IEEE Intl. Symp. on High Performance Distributed Computing, August 1995 3 avoiding false sharing. It is an adaption and re nement of the two-phase idempotent execution strategy [26] among others. Memory updates are collated to assure exactly-once logical execution, and they are transmitted as bitwise di erences, preventing false sharing. This supports e cient implementation of idempotence in addition to other performance bene ts that we shall see later. 2 Previous and Related Work Calypso has its roots in results by us and by our colleagues addressing fault tolerance, parallel program execution on fault-prone and asynchronous abstract machines, and distributed systems [30, 31, 11, 26, 12, 24, 14, 21, 23, 22, 4, 25, 3, 15, 13]. The research leading to Calypso started as formal work which developed provable methods for executing parallel computations, initially on abstract machines with crashfailing processors, and later on abstract machines with asynchronous processors. An outline of a network of workstations-based system for parallel computing based on earlier formal work was presented in [13]. Calypso is an evolution of this design, and is the result of considerable redesign and extensive experimentation on progressively more and more sophisticated implementations. Major new developments in Calypso include: (1) a programming strategy that allows dynamic thread segment declarations, thus providing programming scalability; (2) dynamically evaluated termination condition to increase e ciency; (3) addition of sequential steps to handle external interactions and side e ects; (4) a global management mechanism which uses bu ering, collating of updates and transmits updates as di erences thus allowing arbitrary logical data granularity; (5) memory locality management. However, it does not implement a faulttolerant manager as described in [13], which relied on dispersal and evasion. Among the many systems that directly address parallel computing on workstation networks, some focus on providing message passing mechanisms. Message passing systems closely resemble the underlying hardware in a portable environment. Popular systems include PVM [16], P4 [9] and MPI [18]. The remote procedure call mechanism adds structure to message passing systems and makes programming a little simpler. Concert/C [2] from IBM and DCE-RPC [28] are examples of mature and portable packages. Calypso provides a high-level programming model that relieves the programmer from handling the underlying communication layer. Another class of systems for parallel computing focuses on providing DSM (Distributed Shared Memory) across loosely-coupled workstations. IVY [29] was one of the rst implementations of DSM. Midway [8], Munin [7], and now TreadMarks [27] and Quarks provide a weaker, and sometimes multiple consistency semantics in order to improve performance. In contrast, Calypso provides a simple, and a uni ed programming model that separates logical and execution parallelism. In addition, load balancing is not left for the programmer, rather provided by the system. Linda [10] is a variant of DSM that provides a common global space. Piranha [17] is built on top of Linda and allows workstations to join an ongoing computation as they become idle, and retreat when reclaimed by their owners. In fact, our daemons are modeled after Piranha. But unlike all of the previous systems, Calypso can mask process crashes. There have been several proposals to provide fault tolerance, mostly by augmenting an existing system. They include FT-PVM [32], FT-Linda [5], PLinda [19], and Orca [20]. A notable exception is DOME [1] that incorporated fault tolerance and load balancing form the onset. These systems provide fault tolerance by using well known mechanisms: checkpointing the data, logging messages, and using reliable atomic broadcasts. In contrast, Calypso uses a uni ed set of techniques to tolerate failures and slowdowns. Furthermore, in the absence of failures, there is virtually no overhead. 3 Syntax and Semantics of Calypso Programs that run on theCalypso system are written in CSL. Programs are structured by inserting parallel tasks into sequential programs. We refer to an In Proc. 4th IEEE Intl. Symp. on High Performance Distributed Computing, August 1995 4 Parallel Step Sequential Step Parallel Step Sequential Step Step 1 Step 2 Step 3 Step 4 Figure 1: A fragment of an evolving execution. execution of such a parallel task as a parallel step, and we refer to the sequential execution fragment between two consecutive parallel steps as a sequential step. The execution of a parallel step consists of several concurrent thread segments. Figure 1 illustrates a small fragment of an evolving execution. CSL is C++ augmented with 4 keywords: shared, parbegin, parend, and routine. Informally, the major features and restrictions of CSL are as follows. The address space of CSL programs is partitioned into two disjoint areas: private and shared. Shared variables are declared by: shared f member-list g A parallel step is a new compound statement to express parallelism. A typical form of one such statement is: